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SUMMARY A coding problem for correlated information 
sources is investigated. Messages emitted from two correlated 
sources are jointly encoded, and delivered to two decoders. Each 
decoder has access to one of the two messages to enable it to re- 
produce the other message. The rate-distortion function for the 
coding problem and its interesting properties are clarified. 
key words: multiterminal source coding, complementary deliv- 
ery, joint encoding, separate decoding 

1. Introduction 

Coding problems for correlated information sources 
were originally investigated by Slepian and Wolf [1]. 
Corresponding rate-distortion coding problems [2]-[4] 
and various coding problems (e.g. [5]-[7]) inspired by 
the work by Slepian and Wolf have been considered. In- 
cluding the above studies, the main focus in the 1970's 
was on coding problems with separate encoding (each 
message is separately encoded) and joint decoding (sev- 
eral codewords are sent to a decoder and decoded si- 
multaneously). 

In contrast, since the 1980's, coding problems that 
involve joint encoding (messages from several sources 
are encoded at once) and/or separate decoding (each 
message is separately decoded) have been explored. 
Separate decoding processes have mainly been consid- 
ered in relation to multiple description (e.g. [8]-[10]), 
while joint encoding processes can be seen, for exam- 
ple, in the cascading and branching communication sys- 
tems [11], the triangular communication system [12] 
and multi-hop networks [13], [14]. 

Also, a coding problem that involves joint encoding 
and separate decoding was considered by Willems et al. 
[15], [16]. The coding system models a communication 
network via a satellite. Several stations are separately 
deployed in a field. Every station collects its own tar- 
get data and wants to share all the target data with 
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Fig. 1 Complementary delivery network 



the other stations. To accomplish this task, each sta- 
tion transmits the collected data to a satellite, and the 
satellite broadcasts all the received data back to the 
stations. Each station utilizes its own target data as 
side information to reproduce all the other target data. 
Willems et al. [16] investigated a special case of the 
above scenario in which three stations were deployed 
and each station had access to one of three target mes- 
sages, and determined the minimum lossless achievable 
rate for uplink (from each station to the satellite) and 
downlink (from the satellite to all the stations) trans- 
missions. Their main result implies that the uplink 
transmission is equivalent to the traditional Slepian- 
Wolf coding system [1], and thus the main problem is 
the downlink part. Henceforth we denote the networks 
characterized by the downlink transmission as general- 
ized complementary delivery networks, and we denote 
the generalized complementary network with two sta- 
tions and two target messages as the complementary 
delivery network (Fig. 1). This notation is based on 
the network structure where each station (i.e. decoder) 
complements the target messages from the codeword 
delivered by the satellite (i.e. encoder). Kimura et al. 
investigated a universal coding problem for the com- 
plementary delivery network [17] and the generalized 
complementary delivery network [18], and proposed an 
explicit construction of lossless universal codes which 
attains the optimal error exponent. Also, Kuzuoka et 
al. [19], [20] simplified the coding scheme by introduc- 
ing a concept of network coding [21]. 

The above previous researches considered only the 
lossless coding problem. In contrast, this paper focuses 
on the lossy coding problem. The minimum achiev- 
able rate given distortion criteria and some interesting 
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properties of the niinimum achievable rate are clarified. 
This paper is organized as follows. Section 2 pro- 
vides notations and definitions used throughout in this 
paper. Section 3 investigates the lossy coding problem 
for the complementary delivery network, which includes 
descriptions of the main result and several related prop- 
erties. The main result can be easily extended to the 
problem of the generalized complementary delivery net- 
works, which will be discussed in Section 4. Finally, 
Section 5 provides theorem proofs. 

2. Preliminaries 

Let X and y be finite sets. Especially, for any natu- 
ral number M, we denote Tm = {1,2,- •■ ,M}. The 
cardinality of X is denoted as |Af|. A member of X" 
is written as a;" = (a;i,a;2, ■ ■ ■ ,a;„), and substrings of 
x" are written as x^ = {xi, s^+i, ■ • ■ ,Xj) ior i < j. A 
set of all the probability distributions on X is denoted 
as P{X). A discrete memoryless source {X,Px) is an 
infinite sequence {Xi}^!^ of independent copies of a 
random variable X taking values in X with a generic 
distribution Px £ PiX), namely 

n 

PxA^n ^l[Px{x^). 
i=l 

P{X\Py) denotes a set of all the probability distribu- 
tions on X given a distribution Py 6 P{y)- Namely, 
each member of P{X\Py) is characterized by Pxy G 
P{X X y) as Pxy ^ Px\yPy- A source {X,Px) 
can be denoted by referring to its generic distribution 
Px or random variable X. For a correlated source 
{X,Y), H{X), H{X\Y) and I{X-Y) denote the en- 
tropy of X, the conditional entropy of X given Y , and 
the mutual information oi X and K, respectively. Sim- 
ilarly, for a correlated source (X, y, Z), I{X;Y\Z) de- 
notes the conditional mutual information of X and Y 
given Z. In the following, all bases of exponentials and 
logarithms are set at e (the base of the natural log- 
arithm). Let X stand for a reconstruction alphabet 
that corresponds to a source X to be encoded, and let 
Ax : X X X ^ [0, Ax] be a corresponding single-letter 
distortion function, where Ax < oo. The vector dis- 
tortion function is defined in the usual way, i.e. 



AJ(x" 



n 

n ^-^ 
fe=i 



{xk,Xh). 



3. Complementary delivery 



3.1 Problem formulation 



Definition 1. (CD (Complementary Delivery) code) 
A set {ipn, (pii , (fii ) of an encoder and decoders is a 

( X^ (Y) 

CD code (ri, Mn, Pn , Pn ) for the source {X, Y) if and 
only if 



</?n 


: X' 


xj"- 


*^A/„ 


^i'' 


■■ ^A/„ X y" - 


->X'\ 


^i^^ 


: Xm„ X A-" 


-r, 


Pi"^ 


= E 


AJ(X" 


,^«(A„,r"))^ 


P'P 


= E 


A'5^(y" 


^(2)(A„,x"))" 



A„ = ^„(x",y"). 

Definition 2. (Lossy CD-achicvable rate) 
R is a lossy CD-achievable rate of the source (X, Y) for 
a given distortion pair {Dx, Dy) if and only if there ex- 
ists a sequence \ ln,Mn,Pn i P" ) f o/ CD codes 
for the source (X, Y) such that 

lim sup — log A/„ < R, 

n — ^oo ri 

lim sup pj/^' < Dx, hmsupp^^^ < Dy. 

n — *oo n — 'OO 

Definition 3. (Inf lossy CD-achievable rate) 
R{X,Y\Dx,Dy) = mi{R\R is a lossy 
CD-achievable rate of{X,Y) for (Dx,-Dy)}. 

3.2 Statement of results 

Theorem 1. (Lossy coding theorem for CD code) 
RiX,Y\Dx,Dy) 

[niax{/(X;[/|y),/(y;C/|X)}], 



: mm 

PuiXY^'PcniUlPxY 



where the alphabet U satisfies 

\U\ < \Xxy\ + 2 

and Pcd{1^\Pxy) ^ P{U\Pxy) is a set of probability 
distributions such that there exist functions 0(1) : lA x 

y ^ X and (j}('2^ : U X X —I- y that satisfy 
Dx > i?[Ax(X,0(i)([/,y))], 

Dy > i?[Ay(r,(/.(2)(C/,X))]. 

Several important relationships between Theorem 
1 and previously reported results are presented in the 
following. 

Lemma 1. (Compatibility with the result obtained for 
the lossless coding) 

Suppose that X ^ X , y ~y , Axix, x) ^ if and only 
if X = X and Ay{y,y) = if and only if y ~ y. In 
this case, the inf achievable rate R{X,Y\Dx,Dy) for 
Dx = Dy — is reduced to the minimum achievable 
rate for the lossless coding. 

dof 
RiX,Y) = R{X,Y\Dx ^0,Dy = 0) 

= mSix{H{X\Y),H{Y\X)}, 

which coincides with the result reported by Willems et 
al. [16]. 
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Proof. Note that if the conditions shown in Lemma 1 
satisfy we have 









u 



Lemma 2. (Relationship to the conditional rate- 
distortion function) 



R{X,Y\Dx ^di,DY) 
R(X,Y\Dx,DY = d2) 



Rc{Y\X,Dy), 
RciX\Y,Dx) 



if di > Ax and d2 > Ay, where Rc{X\Y, D) denotes 
the conditional rate- distortion function [22], namely the 
minimum achievable rate when X is encoded and repro- 
duced both with the side information Y to guarantee the 
distortion criterion D. 

Proof. It is sufficient to show that first equation. The 
condition di > Ax impHes that one of the two messages 
does not have to be reproduced. Therefore, the encoder 
(fin sends the codeword only to the decoder (pn , which 
means that the coding rate characterized by the condi- 
tional rate-distortion function is an achievable rate. 

R{X,Y\Dx^di,DY) < Rc{Y\X,Dy). 
On the other hand, we have 

R{X,Y\Dx,Dy) 

> mai^{RciX\Y,Dx),RciY\X,DY)} 

> RciY\X,DY). 



from the result of Theorem 1 . 



D 



Lemma 3. (Relationships to the conditional rate- 
distortion function and Wyner-Ziv rate distortion func- 
tion) 

ina^{Rc{X\Y, Dx), RciY\X, Dy)} 

< R{X,Y\Dx,Dy) 

< ma^{RwziX\Y, Dx), Rwz{Y\X, Dy)}, 

where R\yz{X\Y, Dx) is the minimum achievable rate 
for the coding system called the Wyner-Ziv coding sys- 
tem [2], where X is encoded without any side informa- 
tion and reproduced with the side information Y . 

Proof. The left inequality was shown in the proof of 
Lemma 2. The right inequality was shown by Kuzuoka 
et al. [20]. D 

Lemma 3 indicates that there may be some rate 
losses only for the lossy coding. This property results 
from the auxiliary random variable U included in the 
inf achievable rate R{X, Y\DxtDy)- 



4. Extension to multiple sources 

Theorem 1 considered only two correlated sources. 
However, the theorem can be easily extended to any 
finite number of correlated sources. 

Let X be a set of TV discrete memoryless sources 

X = {x(i),x(2),...,xW}, 

each of which X*^') takes a value in a finite set X'^^^ 
(i € In)- For a given subset S C Xat of source indexes, 
the corresponding subsets of sources, alphabets and its 
members are denoted by 



X^s) 


= {xW|ze5}, 


A-f^) 


- Jl^-W, 




ies 


a;(^) 


= {a-W e A-t'^lie^} 



Similarly, for a given subset S C I^y, the n-ih. Cartesian 
product of X^^i , its member and the corresponding ran- 
dom variable are written as ^"("5)", aj^-^)" and X^'^^", 
respectively. A substring of a;^'^''" is written as x\ 
for i<i. With 5 = Xtv, wc denote X^'^^" == X". Also 
for a given subset S C Z^, its complement is denoted 
by S^=Jn- S. 

Here, we introduce the definition and the coding 
theorem of the generalized complementary delivery code 
which considers multiple correlated sources, multiple 
encoders and multiple decoders. 



Definition 4. (GCD (Generalized Complementary De- 
livery) code) 

A set {fn, 0n ,■ ■ ■ T^Pn ) of single encoder and M de- 
coders is a GCD code 



(n,Af„,{p(f'^)} 



jeXM,ieSj 



for the source X if and only if for any j e Xm and 
i £ Sj <^ Xx 






Ar. 






¥'n(^") 



(X«",^(/>')(A.,X 



(5-)" 



)) 



where ipn is the output of ipn that corresponds to the 
reproduction o/X''^". 

Definition 5. (Lossy GCD-achievablc rate) 

R is a lossy GCD- achievable rate of the source X for 

a given set 

D = {Dj^i}j(zx^j^i(zSj 

of distortion criteria if and only if there exists a se- 
quence 
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{(7i,M„,{p(f"'*)},ei„,,:e5,)}^^^ 



of GCD codes for the source X such that for any j G 
Xm cind i G Sj 

lim sup — log M„ < R, 

n — >oo ri 

lim sup p^^'*) < Dj^i. 

n — >oo 

Definition 6. (Inf lossy GCD-achievable rate) 
R{X\D) = ini{R\R is a lossy 

GCD- achievable rate of X for D\ . 
Theorem 2. (Coding theorem of lossy GCD code) 

R{X\D) 



min max/ I X^'^^^C/ 

Pu\x(iVcD(U\Px)3(^lM 



where the alphabet ti satisfies 

M 

and Vcoii^lPx) ^ 'Pi'^\Px) *s a set of probability dis- 
tributions such that for any j £ 2m and i G Sj there 
exists a function <j>(j^i) '■ U x X^ i ' — > X^"^' that satisfy 



£'j,, > E 



.;,<., (XW,0(,,,)(C/,X(^.^))) 



As a typical example. Theorem 2 can be applied to 
the coding problem formulated by Willems et al. [16]. 
In this coding system, the encoder sends three messages 
X = {X, Y, Z} to three decoders, and each decoder 
has access to one of three messages to reproduce the 
two other messages. Theorem 2 indicates that the inf 
achievable rate for this coding problem is obtained as 

R{X,Y,Z\Di,D2,D3) 
= min 

Pu\xyz(^Vcd{U\Pxyz) 

max{/(Xy; U\Z), I{YZ; U\X), I{XZ; U\Y)}, 

where the alphabet Z/^ satisfies \U\ < \XxyxZ\-^6, and 
VcDiKlPxYz) C V{U\PxYz) is a set of probability 
distributions such that there exist functions 



X. 



13) -M X X ^ Z, 

23) -M xy ^ z, 
X, (/)(32) -Mxz^y 



(^(12) -.U X X 

(^(21) -.u xy 

0(31) -.U X Z 
that satisfy 

Di2 > ii;[Ay(r,0(i2)(c/,x))], 
Di3 >£;[Az(z,0(i3)(;7,x))], 
D21 >£;[Ax(x,0(2i)(t/,y))], 



D 



23 



> 



£;[Az(z,0(23)([/,y))], 
D31 >£;[Ax(x,0(3i)([/,z))], 
D32 > £;[Ay(r,(/.(32)(c/,^))]. 



5. Proof of theorems 

5.1 Theorem 1: converse part 

Proof. 

Let a sequence {(iy9„, ^„ , ^„ )}^i of CD codes be 
given that satisfy the conditions of Definitions 1 and 
2. From Definition 2, for any S > there exists an 
integer ni = ni{S) and then for all n > ni((5), we can 
obtain 



-logM„ < R + S. 
n 

It should be remembered that A^, 
Then, we obtain 



(^„(x",y"). 



n{R + S) 

> logM„ 

> H{An) 

> i/(A„|r") 

= j(X";>i„|y") (■.•A„ = ^„(x",y")) 
== i?(X"|y") - F(x"|A„r") 



= Y.{H{Xk\Yk) - F(Xfc|A„X'=-iy")} 

fc=i 

n 

= J2liXk;A„X''-'Y''-^Y^+,\Yk) 

n 

> J2HXk;AnX''-^Y''~^\Yk). 



fc=l 



Let us define random variables Uk — AnX'^ ^Y^ ^. 
With these definitions, we have 



n[R + 5) > Y,I{Xk;Uk\Yk). 
fc=i 

In a similar manner, we obtain 

n 

n{R + 6) > Y,Iiyk;Uk\Xk). 



k=l 



Here, let J be a random variable that is independent of 
{X,Y) and uniformly distributed over the set X„. We 
define a random variable U = {J,Uj). This implies 
that 

R + 5 

1 " 

> -y^I{Xk;Uk\Yk) 

1 " 
= - H{H{Xk\Yk) - H{Xk\UkYk)} 
n ^ — ' 

k=l 
1 " 

= - y^{H{Xk\Yk) - HiXj\UjYj, J = k)} 



fc^l 
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H{X\Y) - H{Xj\JUjYj) 
H{X\Y) - H{X\UY) 
I{X-U\Y) 



and 



R + 5 > I{Y;U\X). 

Since (5 > is arbitrary, we obtain 

R > meiyi{I{X;U\Y),I{Y;U\X)}. 

We next show the existence of functions 0(i) and 
(/)(2) that satisfy the conditions of Theorem 1. From 
Definition 2, for any 7 > 0, there exists an integer 
"2 = '^■2(7), and for all n > ^12(7), we have 

fe=i 

= -TElAxiX.^Xk)], 
fc=i 

1 " 
Dy+1> -J2E[AY{Yk,f>^^liA^,X-)) 

fe=i 

= -J^ElAyiYkSk) 

n ^ — ' L 



fe=i 



?(») ^- - 



where 01^'^ [i — 1,2, k e Z„) is the output of (p^^. at 
time fc, and 

We note that UkYk contains AnY^ , and UkXk contains 
AnX^ , which implies that Y^j^-,^ (resp. X^_^i) is fur- 
ther needed to generate Xk from UkYk (resp. Yfe from 
UkXk). Here, let us define the distribution Qki,k2 of 
AnX'^^Y''^, namely for any x''' £ X''\ y''^ e y^'^ and 



an 




1) 

€2 l^n: 


^'\/^) 
















def. 


Pr{^ 


„(x",y") = 


On 


X 


"=1 = 


x^ 


i^yfe2 


= 




= 




>; 






Px 


,y, 


(^",y") 






^"'fci+i 


fc2+l 


xj;" 


-^2 











/n 



Also, let Qfc be the distribution of Xk given UkYk, 
namely for any Uk = anX^~^ y*'"^ 

def. QkAan,x^,y^) 



1(1) 



fc (a;fc|wfc,yfc) 



Qfc-i,fc(a«,a;'' i,?/'') 



(2) 
and Ql, be the distribution of Yk given UkXk defined 

similarly. 

^(2). I ^ def. Qk.k{an,x'',y'') 



Further, let us define Y^_^^{Uk, Yk) (resp. X^+i(C/fc, Xk)) 
as a random variable selected to minimize the average 
distortion between Xk and Xk given UkYk (resp. be- 
tween Yk and Yk given UkXk), namely 



,def. 



Yk\,{Uk,Yk)^ 

J2 Qk\Xk\Uk,Yk)Ax{Xk,Xk) 



arg mm 



Xk£X 



XJ!^^(Uk,Xk) 



def. 



arg mm 



Y, Qf\Yk\Uk,Xk)AY(Yk,Yk). 



Ykey 
We choose the functions (/)(i) and (/)(2) as follows: 

<l>ii)k(Uk,Yk) ^^^- ^i^l(A„,y'^*y,!Vi(c/,,y,)), 

0(2)fe(C/fc,Xfe) '^^^^ ^[^liAn,X''*X[\,{Uk,Xk)), 
def 

0(i)(c/,y) = 0(i)j(c/j,r), 

</.(2)(t/,X) "^^^^ (/.(2)j(t/j,X) 

where * is an operator that represents string concate- 
nation. It is easy to see that 



}k,k-i{an,x^,y^ 1) 



E[Ax{Xk,c^(^)k{Uk,Yk))] 
= E [AxiXk^^^iAn^Y'^ *n\^{Uk,Yk))) 

<s[Ax(Xfe,^«(A.,y")) 
= E\Ax{Xk,Xk) 
E[AY{Yk,d^i^2)k{Uk,Xk))] 
< E[AY{Yk,Yk 
This implies 

Dx+-f > -y^E\AxiXk,Xk) 
fc=l 

1 " 

fe=i 
1 " 

fe=l 

= £;[Ax(x,0(i)(c/,y))], 

Dy +J > E[AYiY,cj)^2){U,X))] . 

Since 7 > is arbitrary, we obtain 

Dx > E[AxiX,<p^,j{U,Y))], 
Dy > i?[Ay(r,(/)(2)(C/,X))]. 

It remains to establish that the bound on \U\ spec- 
ified in Theorem 1 does not affect the determination 
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of the inf achievable rate R{X, Y\Dx, Dy)- To do this, 
we introduce the support lemma [23, Lemma 3.3.4]. We 
can see that 

PxY{x,y) = ^ Pu{u)PxY\u{x,y\u), 

lieu 
I{X; U\Y) = H{X\Y) - H{X\UY) 

= H{X\Y)-Y,Pu{u) 



ueu 



^PxY\u{x,y\u)\og 



{x,y)eXy.y 

I{Y; U\X) = H{Y\X) - H{Y\UX) 
= HiY\X)-J2Pu{u) 



PY\uiy\u) 
PxY\u{x,y\u) 



ueu 



Y^ PxY\u{x,y\u) log 



Px\uix\u) 
PxY\uix,y\u) 



(x,y)exxy 

E[Ax{X,^^r)iU,Y))] 

= YPu(u)YPxY\u{x,y\u)Ax{x,(j>(^i){u,y)) 
ueu (x,y)exxy 

> y. Pu [u) N^ min 



ueu yey 



xex 



YPxY\u{x,y\u)Ax{x,x), (1) 



xex 

i?[Ay(y,(/.(2)(f/,X))] 

= YPu{u)YPxY\u{x,y\u)AY{y,(t)(2){u,x)) 
ueu (x,y)exxy 

> y Pu{u) y rnin 
ueu xexv^y 

Y,PxY\uix,y\u)AY{y,y), (2) 

yey 

where Eq.(l) (resp. Eq.(2)) comes from the fact that 
for given letters {u,y) & Uxy (resp. (w, x) € UxX) the 
output of the function (/in) (resp. 4>t2)) can be selected 
so as to minimize the average distortion. We then define 
the following functions of a generic distribution Q G 
V{X X y): 

qi{Q,{x,y)) = Q{x,y), 

q2{Q) =max{q2.i(Q),g2,2(Q)}, 

Q2,l(Q) 

(x,y)exxy ^^ '^' 

92,2 (Q) 



H{Y\X)~ Y,Q{x.y)\og 



J2Q(^^y') 



y'ey 



{x,y)exxy 



Q{x,y) 



93,1 (Q) = E^^sE Qi^^y)^x{x,x) 

yey '•"^'^ xex 

<?3,2(Q) = Y't^vaYQ{x,y)AY{y,y) 
xex y^y yey 

Note that jA" x 3^| — 1 functions are necessary to pre- 
serve the distribution Q{x,y) and 2 functions to pre- 
serve the average distortion characterized by the generic 
distribution Q. From the support lemma, we can find 
a generic distribution a G P{U) such that U C U, 
|W| < I A" X 3^1 -I- 2 and the following equations are si- 
multaneously satisfied: 

Y a{u)qi{PxY\u{-\u), {x,y)) = PxY(x,y), (3) 
ueu 

Y a{u)q2{PxY\u{-\u)) 



ueu 



= max{/(X;C/|y),/(r;C/|X)}, 
Y a{u)q3^i{PxY\u{-\u)) 



leu 



= E"(")E™^2lE PxY\u{x,y\u)Ax{x,x), 
ueu yey ^'^^ xex 

E"(")?3,2(-Pxy|c/(») 
ueu 

= E"(") E '^'^''l^PxY\u{x,y\u)AY{y,y). 
ueu xexv^yyey 

Here, let us define functions (/)^^ : U x y ^ X and 
(j)*,^-. -.U X X ^y that satisfy 

<i>li)W,y) = argminE -PxF|(7(a^,y|w)Ax(a;,^), 
xex xex 

'/'(2)(w:2;) = aigTim\YPxY\u{x,y\u)AY{y,y). 
y^y yey 

With these definitions, we have 

E«(")'73,i(^xy|c/(») = i?[Ax(X,0^,)(t/,r))], 

ueu 

E«(")93,2(Pxyic/(-k)) = i?[Ay(y,0^2)(C/,X))], 

ueu 

and 

Di > S[Ax(X,0(i)(C/,y))] 
> E[Ax{X,rii){U,Y))], 

D2 > i?[Ay(y,0(2)(C/,X))] 

>E[AY{Y,ri2)iU,X))]. 

Hence, 0^n and 0/'2^ satisfy the conditions of Theorem 
1. Further, Eq.(3) implies that there exist a random 
variable U and a joint distribution Ptj^y ^^^^ satisfy 
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a{u)PxY\u{x,y\u) = P~^Y{u,x,y) 

for all {u,x,y) eU x X x y. The new joint distribution 
preserves the distribution Pxy 

Xl-^c/XY^^'^'y) = '^'^{u)PxY\uix,y\u) 
u&A ueu 

= PxY{x,y). 

This completes the proof of the converse part. D 



5.2 Theorem 1: direct part 

We begin by establishing some notation and mentioning 
a few basic facts that will be used hereafter. 

Definition 7. (Set of typical sequences) 

For any 5 > 0, define the set of typical sequences as 



T-{5) = 



-N{x\x^)^Px{x) 



<5yxe X 



where N(x\x'^) stands for the number of occurrences 
of the letter x included in the sequence x". A similar 
convention is used for other random variables. When 
the dimension is clear from the context, the superscript 
n will be omitted, e.g. Tx{S). 

Lemma 4. (Csiszar-Korner [23]) 
For any d > 

PT{X^eTxi5)} >l-e^{S), 

where 

lim e„((5) — 0. 

n — >oo 

Lemma 5. (Csiszar-Korner [23, Lemma 1.2.10]) 

For any 6,S' > 0, if (a;",y") £ Txy{Si) then x" £ 

TxiSi\y\). 

Lemma 6. (Steinberg-Merhav [24]) 
For any 6' > 6 > and x" e Txi6) 

exp{-n{I{X;U) + ei)} 

< Y. Pu{un<GM~n{IiX;U)-e2)}, 

where ei is a function of (5,5'), €2 is a function of (5,5') 
and 

lim ei = lim £2 = 0. 

Now, we proceed with the proof of the direct part 
of Theorem 1 . 

Proof. 

Let a distortion pair (Dx, Dy) be given, and Pu\xy G 
'Pcd{U\Pxy)- Fix arbitrary 7, (5 > 0. 



Codeword selection: ip„ 

(1) Randomly generate Mjj independent codewords 
u"(j) G Z//" (i S Imu)^ each of length n, according 
to Pjj to create a codebook Au = {w"(«)}i='i- 

(2) Partition the codebook Au into Nu bins, each con- 
taining Lu = Mu /Nu members of Au- For simplicity, 
Mu is a multiple of Nu. Let Au{j) denote the subset 
of Au whose elements are assigned to bin j (j G ^Nu ) ■ 
Without loss of generality, we define 

Auij) = K(^)K;^_i)^„+i. 

Encoding: fn 

(1) For a given input pair (a;",y") G X" x y^ of se- 
quences, the encoder seeks a vector u" G Au that sat- 
isfies (M",a;",?/") G TuxY(ki5), where fci > 0. If there 
is more than one such vector in the codebook Au, the 
first one is chosen. If there is no such vector in the code- 
book Au, a default vector is chosen, say m"(1), and an 
error is declared. The selected vector is denoted by 
M"(a;",j/"). 

(2) The value assigned to the encoder ipn(-) is the bin 
index to which u"(a;",y") belongs, that is, 

^„(x",y") = J ifu"(x",2/")G^[/(j). 

Decoding: (pn 

(1) The decoder has access to the bin index ju G Inu 
received from the encoder and the sequence y" G 3^" of 
side information. 

(2) The decoder seeks a unique vector w" G Au(iu) 
that satisfies (u",j/") G TuY(k25), where k2 > 0. This 
vector is denoted by u"(y^). If there is no or more 
than one vector u" G Au(ju) jointly typical with y", 
arbitrary u" is chosen, and an error is declared. 

(3) The reconstruction vector x" = (xi,iE2, • • • ,Xn) is 
given by 

Xk = (/)(i)(ufe(2/"),yfe) (fc G Xn), 

where Uk(y^'') is the k-ih. element of u"(y"). 
Decoding: (pn 

(1) The decoder has access to the bin index ju G Inu 
and the sequence a;" G X"^ of side information. 

(2) In a similar manner to ^n , the decoder seeks a 
unique vector u" G Au{ju) that satisfies (u",a;") G 
Tux(kz5), where fcs > 0, and the reconstruction vector 
y" is given by 

yk = <t>{2)[uk[x'^),Xk) [k G In). 

Distortion evaluation: ifn 
For the distortion, we obtain 

A^(x",x") 



1 

- y^ Ax(xk,xk) 

n ^ — ' 

k=l 



1 " 

= -Y'!^x(xk,(l>{i){uk(y"),yk)) 



fc=i 
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= 1 V iV(^,z,y|S"(y"),x",2/") 
n ^-^ 

{u,x,y)eUxXxy 

^x{x,(t)(i){u,y)). 

We note that (M"(a;",y"), a;",?/") S TuxY{ki5). Also, 
if no error occurs in the encoding/decoding process, we 
have M"(x",y") = u^{y'^). In this case, the following 
inequalities are satisfied: 

< ^ {PuxY{u,x,y) + ki5)/\x{x,(j)(i){u,y)) 

(u,x,y)eUxXxy 

< ^[Ax(X,0(i)(t/,r))] + kiSAxlU X X X y\ 

< Dx+ki5Kx\Ux X xy\. 

We denote error probabilities in the encoding/decoding 
process as P". Then, the average distortion can be 
bounded as 



E 



A^(X",X") 



< (1 - P2){Dx + ki6Ax\U xX xy\) + P^Ax- 

Since (5 > is arbitrarily small for a sufficiently large 
ri, if P" vanishes as n — > oo, we can obtain 



limsupP A^(X",X" 



(2) 



Distortion evaluation: (p. 
We can obtain 

limsupP[A?,(r",y") 



< D, 



< Dy 



in a similar manner to cpn . 

Error evaluation: tpn 

If there is no u" S Au that satisfies (u",x",?/") e 
TuxY^kiS), an encoding error has occurred. This event 
is denoted as 



def. 



Mu 



Ell = n {(«"W,a;",2/") i TuxvihS)} . 

i=l 

Here, let us define 

Po'^^^- {(.T",y")erxy(M)}, 

where fco > 0. From Lemma 4, PrlPg} -^ as n — > oo. 
Then, wc have 

Pr{Pi} < PrjPiUPo'} 

= Pr{Po=} + Pr{PonPi}, 

PrjPonPi} 
< Y. PxYix^yn 

{x",y'^)eTxY{koS) 
( Mu 

Prl f]{{U"iz),x",y'^)iTuxY{hS)} x^,y- 
[i=i 



J2 PxY{x\yn 

(x'^,y")GTxY{koS) 

(Mu 1 

Pvl fl {{U"{t),x",y-) i TuxYihS)} \ 

{'.' u"'{i) is selected independently of (a;",y")) 

< Yl PxYix^\yn 

{x^,y")&TxY(koS} 

[l-cxp{-n(/(Xr; [/) + £„)}]*'- 
(■.• Lemma 6) 

< Y PxYix-.yn 

{x^,y"')GTxYikoS) 

exp [-Mu exp {-n{I{XY; U) + e„)}] , 
(■.• (1 - a)" < cxp(-an)) 

where e„ is a function of {kiS, kaS). By setting Mjj, fci 
and fco as 

Mu > exp{n{I{XY; U) + mi7)}, toi > 0, 

mi'^ > e„ and fci < fcg, we have lim„^oo PrlPi} = 0. 

Error evaluation: (pn 

If there is no or more than one u" g ^t/OV) such that 
{u^,y^) G TuYik^S), a decoding error is declared. This 
event is classified into two cases. 

(1) First case: (u"(x",y"), y") ^ TuY{k26). However, 
this error docs not occur by setting fe as fc2 > ki\X\ 
because (M"(a;",?/"),a;",?/") S P(7XF(fci'^) and Lemma 
5. 

(2) Second case: If there exists u" G Au{ju)i u" ¥" 
u"(x",y") such that (u",y") G P(7y(fc2^)- This event 
is denoted as 



E, 



dcf. 



U {K,y")Gr^y(fe5)}. 

M" e^u Uu ) ,M" #M" (a" ,'y" ) 



Let i(j, fc) be the index i oi k-th u"(z) in Au{j), namely 
from the definition of Au{j) we have 

i{j, k) = U - l)Lu + k. 

Since if (x",?/") G TxY{ko5) then y" G Ty (/cq^IA-I), we 
have 

PrjPa} < Pr{P2 U Pg} 

= Pr{Po"} + Pr{PonP2}, 

PrjPonPz} 

<i: T. Priy") 

k=y"£TY{koS\x\) 

Pr{(P"(*(jt,,A:)),y")GTt,y(fc2<5)} 
(•.• u"(z) is selected independently of y") 
<ic;exp{-n(/(F;P)-e)}, 
(■.• Lemma 6) 
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where e is a function of (fco(5||A:'[[, A;2'^)- By setting Ljj 
and fc2 a-s 

Lu < exp{n(/(y; U) - /17)}, h > 0, 

Ii7 > e and fco|A:'| < k2, we have hm„^oo Pt^IE-i} = 0. 

Error evaluation: ipn 

This is almost the same as the case of (pii . We have 
to set 

Lu < exp{n(/(X;[/)-?27)}, ^2 > 

to vanish the encoding/decoding errors. 

Rate evaluation: (/?„ 

The encoder sends the indexes of the bin using 

R^-\ogNu 
n 

1 Mu 

= -log^— 

n Lu 

> I{XY;U)+mi-f 

- min{/(y; U) - l^, I{X; U) - hi] 
= max{I{X; U\Y) + hj, I{Y; U\X) + hj} + rmj 

bits per letter. Since 7 > is arbitrary, we obtain the 
coding rate as max{/(X; U\Y),I{Y; U\X)}. 

This completes the proof of Theorem 1. D 

5.3 Theorem 2: converse part 

Proof. 

The proof of Theorem 2 is quite similar to that of The- 
orem 1. Let a sequence {((/?„, ^„ ,■' ' if'n )}^i of 
GCD codes be given that satisfy the conditions of Def- 
initions 4 and 5. From Definition 5, for any S > there 
exists an integer rii = ni{S) such that for all n > ni((5), 
we can obtain 

- log Mn < R + S. 
n 

In a similar manner to Theorem 1 we obtain 

n 

n{R + 6) > ^/(Xf ^A„X^-i|xf ^ 

k=l 

Let us define random variables Uk = AnX '~^, and 
let J be a random variable that is independent of X 
and uniformly distributed over the set X„. We define 
a random variable U = {J,Uj). This implies that for 
every j e Xm 

R + S > I{X'-^'^;U\X^^^^). 

Since (5 > is arbitrary for a sufficiently large ri, we 
obtain 

R > mi^xI{X^^''>;U\X^^^'>). 



We next show the existence of functions 



'UA 



O'G 



Xm, i & Sj) that satisfy the conditions of Theorem 2. 
From Definition 5, for any 7 > there exists an integer 
712 = "-2(7) such that for all n > 712(7) 



1 " 
n ^-^ 



[a^«(x«,^2^)(a„,x(^^)"))^ 



■(i\i 



where (^„ ';. [k £ Z„) is the output of (pn' at time k. 
We note that UkX[ ' contains A^X^^^^'' , which im- 
plies that Xf^J^^ is further needed to generate Xf. ' 

from UkXf, ' . Here, let us define the distribution 
Qh^M of A^X'^^^^^^X^^'i'^^^ namely for any a;('5.)fci e 
X{Si)k^^ xiS-)k2 g ^(S,=)fc2 and a„ G Zm„ 

"^^^^ Pr{^„(X") = a„, 

\^ fcl+1 ■ fc2 + l J 



Also, let Qjf be the distribution of X^, ^ given 
C/fcX^ ' , namely for any Uk = anX^^'^^''^-^ 

Q^^\x['^^\uk,xf^) 



dcf. gfc,fc(a„,a;(^g^^)^a;(^g.')^) 



Further, let us define Xf._^i {Uu^Xf, ' ,i) as random 
variables selected to minimize the average distortion 
between X^' and the output of ^„ fc (« £ Sj) given 

UkXj^ ' , namely 

-^(S^Jn (5J) def 



x:^"e^<^i'"- 



arg inm 

5: g^)(xf^)|f4xf') 

A^«(X«,^^^)(A„,X(^^)")). 
We choose the functions <p{j-i) as follows: 



4'{r,i)k{Uk,Xk 



(5J) 



def. ,-^(j;i) 



V'. 



(SJ)«, 



!^f(A„,x(-^)'^*xr:^r(c4,xr^\z)), 



.(5;) 



.def. 



In a similar way to Theorem 1, we obtain 
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Dj., + 1 

k=l 
1 " 



Since 7 > is arbitrary for a sufficiently large n, we 
obtain 



Dj,, > E 



A;,(„(X«,0(,.,)(C/,X(^.^)))' 



It remains to establish that the bound on \U\ spec- 
ified in Theorem 2 does not affect the determination of 
the inf achievable rate R{X\D). In a similar way to 
Theorem 1, we then define the following functions of a 
generic distribution Q G P{X^-^'^^): 



q2{Q) = maxq2.j(Q), 



^ Q(^^^x^^^^) 



- J2 Q(a'^^"^)log 



-(^.igA-'^j-) 



Q(a;(^«)) 



l3,m{].i){Q) ~ / , 






=r(^j>eA'(^3> 



where j £ Xm i G 5j and rn{j, i) denotes the serial 
number of the source Xj contained in the index set 5, 



defined as follows 

, . .. def 
m{j,i) = 



i-i 

{teS,\i<i}\+J2\S,\- 

7=1 

Note that |A:'(-^")| — 1 functions are needed to preserve 
the distribution (5(a;^-^™)), and X)j-i \^j\ functions to 
preserve the average distortion characterized by the 
generic distribution Q. From the supporj; lemma, we 
can find a generic distribution a G ViU) such that 
UCU, 

M 

\u\ < \x^^"^\ + J2\s,\ 
J=l 

and the following equations are simultaneously satis- 
fied: 

Y.^a{u)q^{Px\u{-\u),x^^"^) = Pxix^^'^^), (4) 
ueu 



^a{u)q2{Px\ui-\u)) 



ueu 



maxI{X^^^^;U\X^^^'>), 



ueu 



Y^ aiu)q3,rnU,t)iPx\ui-\u)) 
u 
y a{u) y min 



x'^'i'ex"'^' 



Here, let us define functions (j)*,.,^ : U x A'^'^j' — > X^"^^ 
{j G Xa/, i <E Sj) that satisfy 



ir,^) 



4>*^^^.^^^{u,x^^l>) = argmin 
With these definitions, we have 

E "('")'?3,m(j,4)(-Px|C/(») 



new 



i?[Ax(,(xW,0T..([/,x('5.^)))] 



and 



Dj., >i?[A^„(X«,0(,^,)([/,X(^.^)))] 
>i?[Ax(„(X«,0^^.^,)(C/,X(^.^)))]. 

Hence, 4>*i--i\ satisfies the conditions of Theorem 2. Fur- 
ther, Eq.(4) implies that there exist a random variable 
U and a joint distribution PTjy. that satisfy 

a{u)Px\u{x'^^''^n) = P~^{u,x^^^'^) 

for aU (u,£c(^")) G Z? x A'^^"). The new joint distribu- 
tion preserves the distribution Px 



E^^x("'^'^"') = E"(")^^it/(^ 



(Xa 



ueu 



ueu 



Pxix^^^^). 



This completes the proof of the converse part of 
Theorem 2. D 



5.4 Theorem 2: direct part 

Proof. 

The proof of Theorem 2 is quite similar to that of The- 
orem 1. Let a set D of distortion criteria be given, and 
Pu\x e Vcd{U\Px)- Fix arbitrary 7, (5 > 0. 

Codeword selection: ipn 



The same way as Theorem 1. 
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Encoding: tpn 

Almost the same way as Theorem 1. 

(1) For an input set a;^-^™^" S A:'^-^")" of sequences, 
the encoder seeks a vector w"(i) g Au such that 
(w"(i),a;(^")") € Tux{ki5), where fci > 0. The se- 
lected vector is denoted by w,"(a;(-^™^"). 

(2) The value assigned to the encoder (^„(-) is the bin 
index to which u"(a;(-^™'") belongs, that is. 

Decoding: cpii 

Almost the same way as Theorem 1. 

(1) The decoder has access to the indexes ju received 
from the encoder (p„ and the sequence set x^^^'" £ 

(2) The decoder seeks a unique vector w" S Au{ju) 
such that (u", a;('5j")") G T (sp {k2,j5), where fczj > 0. 
This vector is denoted by u^{x^^i'^^). 

(3) The reconstruction vector x^ ^'" is given by 



-(i;j")n 



s(^u) 






where Uk{x^^i^") is the fc-th element of u"(a;('^i^") 

Distortion evaluation: ipn 

In the same way as Theorem 1, we obtain 

(M,2;(^«))gWxA'(^Jv) 



< ^ 



+fci,5Ax(.) \U X A-f^"^ 



We denote error probabilities in the encoding/decoding 
process as P". Then, the average distortion can be 
bounded as 



E 



A^(.,(X(')",X(^'J>) 



< (1 - P^)iD,,, + A;i(5A^(., \U x X<^^«^\) + P^A^,.-,. 

Since (5 > is arbitrarily small for a sufficiently large 
71, if P" vanishes as n — > oo, we can obtain 



lim sup E 



AJ(.)(XW",X(^'^)") 



< D 



j,«- 



Error evaluation: ipr, 



If there is no m" G Au such that (w",a;(^")") G 
Tux{kiS), an encoding error has occurred. This event 
is denoted as 

i=l 

Here, let us define 

Eo'^^^- {(a;(^"'")eTx(M)}, 

where ko > 0. From Lcnuna 4, Pr{_BQ} ^ as n — > cxd. 
Then, in a similar manner to Theorem 1, we have 

Pr{Ei} < Pt{EiUE^} 

= Pv{E'^} + Pv{EonEi}, 

Prison Pi} ^0 (n^oo) 
by setting M{j, fci and ko as 

Mu > exp{n(/(X; U) + mi-/)}, toi > 0, 
iTT-il > £« ~ ^u{ki5, k^5) and fci < fco. 

Error evaluation: ipii 

If there is no or more than one uV'^-, G Auiju) such 
that (uj\, a;^'^j'") G T (s<?)(fc2^), a decoding error is 
declared. This event is classified into two cases. 

(1) First case: 

However, this error does not occur by setting fc2 a-s fc2 > 
fci I A'^'^^) I because 

(ii"(a;(^")"),a;(^")") G Tc/xCfci'?) 

and Lemma 5. 

(2) Second case: If there exists u" G Au{ju)i """ ¥" 
u"(a;(^")") such that (m", a;*'^^''") G T (spik2S). 
This event is denoted as 

P2 '^=- U K^",^*^^^^") e r^^(s|,(fc2'5)}. 

Note that if (a;(2:iv)«) ^ Tx{ko5) then 

^(s;)n g T^(5e)(fco^|A'('5^)|). 

Therefore, we have 

Pr{P2} < Pr{P2UPS} 

= Pr{Po^} + Pr{PonP2}, 
Pr{PonP2} ^0 (n^oo) 

in a similar manner to Theorem 1 by setting Lu, fc2 as 

Lu < exp{n(/(X('5;); [/) - hjl)}. h > 0, 
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hjj > e = e{koS\X<-^^^,k25) and fcol-^"^"^^^! < ^2- 



Rate evaluation: ip„ 



The encoder sends the indexes of the bin using 

R=-\ogNu 
n 

1 Mu 
n Lu 
> I{X; U) + mi7 - min {/(X^'^^'^; U) - /y7} 

= max{/(X('5^'; [/IXt-^^')) + hj-f] + mi7 

bits per letter. Since 7 > is arbitrary, we obtain the 
coding rate as maxje^^. I{X^^^'^;U\X^^'^^). 

This completes the proof of Theorem 2. D 
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